feat: add built-in fetch skill (read webpage text) - #132
Open
hallerite wants to merge 1 commit into
Open
Conversation
hallerite
force-pushed
the
feat/fetch-skill
branch
from
August 28, 2026 16:51
050654f to
254397b
Compare
hallerite
marked this pull request as ready for review
August 28, 2026 16:53
hallerite
force-pushed
the
feat/fetch-skill
branch
from
August 28, 2026 17:35
254397b to
845cd87
Compare
snimu
reviewed
Aug 28, 2026
| "several angles at once, fan out with `asyncio.gather(search(...), search(...))`." | ||
| ) | ||
|
|
||
| FETCH_SKILL_PROMPT = ( |
Contributor
There was a problem hiding this comment.
why is this prompt here? other skills are given with a description in a generic way, i don't understand why this is so privileged. not a fan of such special cases unless there's a strong need.
mikasenghaas
previously approved these changes
Aug 28, 2026
hallerite
force-pushed
the
feat/fetch-skill
branch
from
August 29, 2026 23:21
845cd87 to
f785f40
Compare
Pre-imported async `fetch(url=...)` skill: httpx GET -> cleaned text (scripts/styles/tags stripped), UTF-8-safe decode, truncated to max_chars. A neutral prompt line advertises the skill and its signature when enabled, mirroring the existing edit/search skill lines. Pairs with `search` (search finds URLs, fetch reads them).
hallerite
force-pushed
the
feat/fetch-skill
branch
from
August 30, 2026 13:51
f785f40 to
9c5594d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a built-in
fetchskill: pre-imported asyncfetch(url=...)that GETs a page with httpx (redirects followed, browser-ish UA) and returns its cleaned text — scripts/styles/templates stripped, tags collapsed, entities unescaped — truncated tomax_chars(default 20k). Non-UTF-8 pages decode safely (errors="replace"). Network/HTTP errors come back as shortError: ...strings (data for the agent, not a crash).Pairs with the
searchskill:searchfinds URLs,fetchreads them. A neutral prompt line advertises the skill and its signature when enabled.src/rlm/skills/fetch.py— the skillsrc/rlm/skills/__init__.py— registersfetchin the built-inssrc/rlm/prompt.py—FETCH_SKILL_PROMPT, emitted when the skill is enabledREADME.md— documentsfetchhtml_to_textmarkup strippingMeasured effect (paired A/B, redsearcher, 100 tasks/arm, same pin, skills-only diff, no sub-agents)
*capped = per-session 100 turns, 100k context, 8k/call, 10k tool-out (identical across arms).
Reading: no significant reward gain from the skill itself. Mechanism from mining the control arms: a strong code model simply substitutes — deepseek hand-rolled 12,086 page reads (
requests.get/urllib/curl) across 100/100 control rollouts, sofetchis a convenience wrapper for a capability the model already exercises. Laguna shows a small positive trend (+0.042, n.s.) with modest adoption (29/100 rollouts, 391 calls). The value offetchis therefore primarily hygiene (clean text instead of raw HTML in context, uniform error strings, one sanctioned path that a truncation policy can target) rather than measured task reward.Failure taxonomy from the runs (follow-up material)
2,971 fetch calls (deepseek arm): 33% error rate, dominated by
404(835× — 631 on en.wikipedia.org: the model guesses/wiki/TitleURLs from memory), then 403 (71), 429 (34). 28% of successful reads hit the 20kmax_charscap. Suggested follow-ups:search()to find the real URL") or auto-fallback to the site's search endpoint for known hosts (Wikipedia opensearch).fetch(url, start_char=N)or a trailing "N more chars available" note) so truncated reads can page through.Retry-Afteron 429.Also planned per review: expose fetch as a tool as well as a skill.
First of a 4-PR stack: fetch (this) → contract-only config (#160) → execution guardrails (#158) → role-aware sub-agent prompting (#151).
Note
Medium Risk
Enables outbound HTTP from the IPython kernel when the skill is turned on, which expands attack surface and context size (large page reads), though behavior is bounded by truncation and matches existing network-capable skills like bash/requests.
Overview
Adds an opt-in
fetchbuilt-in skill viaRLM_SKILLS, alongsideeditandsearch, so agents can callawait fetch(url="...")from IPython instead of ad-hoc HTTP/curl.The new
rlm.skills.fetchmodule GETs pages with httpx (redirects, timeout, default UA), normalizes scheme-less URLs, converts HTML to plain text (strip script/style blocks and tags, unescape entities), truncates at 20k chars by default, and returns short error strings on failure rather than raising. Registration writes the usual session stub like other kernel skills; the system prompt gainsFETCH_SKILL_PROMPTwhenfetchis enabled. README documents the skill andRLM_SKILLSvalue. Tests cover stub generation andhtml_to_textmarkup stripping.Reviewed by Cursor Bugbot for commit 9c5594d. Bugbot is set up for automated code reviews on this repo. Configure here.